Files to consider

13GB Crime (Use sample to build) LSOA_popv2 -> Geo + city LSOA(2011)_toLOSA -> Geo + city location

Big Data Product: Weapons and Drugs

In the television documentary “Ross Kemp and the Armed Police” broadcast 6th September 2018 by ITV, multiple claims were made regarding violent crime in the UK.

These claims were that:

1. Violent Crime is increasing
2. There are more firearms incidents per head in Birmingham than anywhere else in the UK
3. Crimes involving firearms are closely associated with drugs offences

To solve this problem, you will use publicly available data sets that have been prepared for you and placed online. These include (but are not limited to):-

1. Street Level Crime Data published by the UK Home Office, this dataset contains 19 million data rows giving a crime type, together with their location as a latitude and longitude.
2. English Indices of Deprivation Data: The English Indices of Deprivation 2010 data set contains the rankings of measures of deprivation within small area level across England. The 32000 localities are ranked from the least to most deprived, scored on seven different dimensions of deprivation.

Assignment Specifics

1. Process the given data efficiently using Apache Spark on a cloud Infrastructure as a Service (IaaS) platform. A sample Jupyter Notebook has been provided on Blackboard.
2. Filter the dataset so that only relevant crimes are included.
3. Using appropriate techniques, determine whether Violent Crimes are increasing, decreasing, or are stable.
4. Determine whether there are more firearms incidents per head in Birmingham than anywhere else in the UK. Possession of firearms carries a mandatory prison sentence in the UK. Therefore, you may assume that a crime type of “Possession of weapons” whose outcome is “offender sent to prison” was a firearm incident.
5. Using appropriate techniques, determine whether firearms incidents are associated with drugs offences.
6. Select and prepare no more than four visualizations to support your analytic findings from (3).
7. Explain the reasoning behind your code so that it is clear what each block is intended to achieve (i.e., appropriately comment the command line).
8. Assess the three claims given and determine whether they are true, false, or cannot be determined.
9. Critically assess and report on the advantages, disadvantages, and limitations of the methods used.
10. Your submission will be a Jupyter Notebook containing both code (typically Python), and explanatory text (in Markdown format) limited to 2500 words (plus references).

1. Introduction

The Crime Analysis task and Approach taken to the problem

Crimes in general is a huge problem for society, peace of mind, locality and affects various parts of life for many people.

Introduction

• Drugs were initially made for medical purposes. They have short & long term effects in the body both physically & mentally.
• Criminals started selling drugs illegal to civilians turning them into addicts making them recurring buyer.

• Since drugs are illegal, people pay extra money to the dealer / criminal as a fee for security & shipping.

• Weapons are illegal to own in various countries.

• People who generally own weapons are cops / securities / person with proper permission under various restrictions.
• When crimes involve both weapons and drugs, they are always nuisance to innocents who caught in the act, public & society.

Approach taken

• Using Big data processing tool (PySpark) running in cloud infrastructure IAAS (like AWS ec2) or PAAS (like AWS Glue or Azure Data Factory)
• The analysis is to study crimes happening in United Kingom (UK) between time period 2010 to 2020
• Using various statistical & visualization techniques we study trends and assert various claims by media.
• How are we doing the analysis?
  1. Use PySpark to process & crunch the data for better handling of raw data (~19MM rows)
  2. Look at various visual lenses to translate the data into into much more digestable
  3. Apply statistical methods to help to test various hypothesis & claims

Difference beween IAAS and PAAS

2. Component Selection and Data Pipeline Implementation

3. Data Extraction and Filtering System running, test and diagnostics

Given Specifics:

1. Filter the dataset so that only relevant crimes are included.

4. Design, Development and reasoning behind use of multiple visualization methods, statistics, and machine learning Models

1. Using appropriate techniques, determine whether Violent Crimes are increasing, decreasing, or are stable.
2. Determine whether there are more firearms incidents per head in Birmingham than anywhere else in the UK. Possession of firearms carries a mandatory prison sentence in the UK. Therefore, you may assume that a crime type of “Possession of weapons” whose outcome is “offender sent to prison” was a firearm incident.
3. Using appropriate techniques, determine whether firearms incidents are associated with drugs offences.
4. Select and prepare no more than four visualizations to support your analytic findings from (3).

General View

Crimes across Year-Month

Observation:

* Overall crimes in UK have a cyclic pattern with peak in mid year
* Overall crime total decreased during 2014 to 2016 comparing to rest of the years

Observation:

• Other Crimes initally seems to be miscategorized
• Violence and sexual offences is increasing over time
• Anti-soical behaviour trend keeps decreasing till 2019 and suddenly increases in 2020. Probably due to outrise in corona
• Drugs have a constant trend over time
• Violent Crime has a declining trend. Data is not available after Apr-2013
• Pattern of Shoplifting, Burglary, Vehicle crime & Criminal damange and arson are similar
• If Violent Crime and Violence and sexual offences are combined together, there is an upward trend. As of 2021, Crimes increased 3x times of 2013

UK Geograpy - Density Heatmap Observations

• Crimes are more in Counties
• Crimes are highly dense in London & Birmingham

Finding Crime that relate to Weapon & Drugs

Checking Various Crimes Types

Grouping crimes types together and taking count of crimes

Observations:

• For weapons, following types can be used: Possession of weapon, public disorder and weapons
• Drug crimes is bigger than weapon crimes
• Anti-social behaviour is 2x of Violence and sexual offences

5. Selection, application, and reasoning behind use of statistical analysis and multiple evaluation measures

Media Claim #1 " Violent Crime is increasing "

To prove the claim,

1. Filter for CRIME_TYPE='Violent crime' or 'Violence and sexual offences'
2. Look at trend over time
3. Additional insight: Look at trend over time by LSOA Area

Media Claim #2 "There are more firearms incidents per head in Birmingham than anywhere else in the UK"

To prove the claim, preprocessing is required

1. Make firearm dataset: Filter for CRIME_TYPE = 'Possession of weapons'
2. Aggregate population by Area name
3. Aggregate Crimes by Area name
4. Join step 2 & step 3 by Area name
5. Calculate ratio between FIRE_ARM_PER_HEAD= $#of firearm cases / #of population in LSOA area$ and PCT (Percentage)
6. Create rank index based on descending order of ratio. (Lower rank or higher ratio = more fireamrs per head)
7. Do a trend chart to between Birmingham and top 20 rank
8. Check Birmingham rank over time

Observations:

• Based on number of crime reports across years, Birmingham is 2nd
• But number of fire arm incidents per person, Birmingham is not the leader

_Note: FIRE_ARM_PER_HEAD_RANK will be equal to FIRE_ARM_PER_HEAD_PCTRANK rank because PCT is multiplied by 100 .

Media Claim #3 "Crimes involving firearms are closely associated with drugs offences"

To prove the claim,

1. weapon_and_drugs = (CRIME_TYPE='Possession of weapons' | CRIME_TYPE='Public disorder and weapons' | CRIME_TYPE='Drugs')
2. Combine 'Possession of weapons' + 'Public disorder and weapons' together as 'WEAPON_CRIME'
3. Visualize trend of weapons & drugs
4. Statistical test: Correlation
   • closer to 1 or -1 means weapons/drugs are always related with drug/weapons crimes
   • Equal to 0 means weapons are not related with drug crimes
   • For the test, we will use >0.7 and <-0.7 correlation threshold to conclude if weapons & drugs are correlated

Observations

• Trend of weapon crimes did not change much, it is flat and did not shift its trend much
• Trend of drugs starts going down in 2014 till 2018
• By looking at the correlation score (0.54) it is correlated but not enough to conclude the statement "Drugs are associated with Weapons" as it did not cross (0.7) correlation threshold

Observations

• There are 5 Areas which have correlation >0.7 indicating areas having weapons are involved in drug related:
  • Areas: Birmingham, East Riding of Yorkshire, Manchester, Salford, Stockport

Burglary Protection

Client: Insurance company developing a highly segmented home insurance product

Claims to prove:

1. There are more burglaries in more affulent areas
2. Burglaries are increasing, decreasing, or are stable

To prove the claims:

1. There are more burglaries in more affulent areas

1. Approach: Rank the Area names based on total #of Burglary crimes over years ## Observations:

    1. Leeds, Birmingham, Sheffield, Bradford, Liverpool, Bristol, Manchester are top areas where the Burglary crimes trends are high compared to other areas
    2. We can create buckets / bins based on the ranks / #of crimes

2. Burglaries are increasing, decreasing, or are stable

1. Approach: Look at the trend of burglaries over time ## Observations:
   1. Cyclic trend of Burglary crimes is consistent & peaks during end of the year.
   2. Between Jan 2014 and Mar 2019, the trend is $\pm$ 3K crimes from the mean of 35K crimes
   3. Burglary reduced by 70% comparing Jan 2020 and March 2020 due to Covid-19 based on Timeline of UK government coronavirus lockdowns and measures, March 2020 to December 2021 ## Conclusion: #### 1. Forecasting of Burglary crimes are significantly skewed due to Covid-19 effect. #### 2. If we negate the covid-19 period (2019 FY - 2021 FY), Burglary crimes started declining and became stable after Jan 2014. #### 3. Across the timeline with crime peaking during end of Year

1. Aggregate by area names and look at their trend

Takeaway for the Insurance Company

Use of bucketing technique, we can create clusters containing areas which can be used to set policy price.

Observation:

1. Using binning technique, 10 Burglary crimes can be bucketed into 10 groups based on #of Burglary crimes across years. Looking at the #of areas in the bucket, after 4th bucket we can combine everything into 1 group

Better approach instead of Binning

A better technique which can be used is K-Means Segmentation.

1. Create various features for the model
2. Score the areas
3. Between each model score runs, track how much data points are shifted from 1 segment to another
4. When the clusters are shifted drastically (say 10%-20% across clusters) then re-run K-Means model again

Using Machine Learning To Forecast Crimes

Using Regression model to forecast crimes related to (Violen crimes, weapon crimes & drug crimes)

6. Detailed Analysis and consideration of the appropriateness of the solution for the initial problem

Observations:

• Filter:We are narrowing down to 'Possession of weapons' & 'Drugs' for forecasting
• Forecast to future: 12 Month window
• Machine Learning Model: Linear Regression

Model Performance:

• The regression model was able to predict as close as possible
• We can also try with various other regressor models like xgboost, catboost, lightgbm and so on.
• Another approach is to go with Time series models like fbprophet, ARIMA, SARIMA models

Feature engineering:

• When using Regression model, we need to create lags of target variable.
  • Using lags, we can introduce seasonality in the data.
  • This is one of the way, time series models use to create relationship behind the seasonality pattern

How to use Regression for time series: | Fix: Remove

1. Select data:
   • Data usually contains 2 columns: timestamp (month) & Target (crimes)
2. Create features:
   • Using lag, we create a shift up to desired level so they help in seasonality
3. Drop nulls
4. Use lags as input for the model
5. Train the model and predict
6. Forecasting:
   • Start from known target
   • Once the forecast is done, insert new row and shift lags by 1 so it becomes input for next month

7. Evaluation and Conclusion

Conclusion of Claims by Media:

1. Violent Crime is increasing - Media:

   • Conclusion:
     • Violent Crimes are increasing as per the data. The statement is _True_ - Analsyis
     • Based on trend analysis over months & year, we can conclude that violent crimes are increasing
     • As of 2021, violent crimes has increased by 3x of 2013

2. There are more firearms incidents per head in Birmingham than anywhere else in the UK - Media:

   • Conclusion:

     • We can draw 2 conclusions here,

       1. Based on #of Firearm incidents reported, Birmingham is the 2nd highest reported</trong> & 1st position is crimes reported in unknown location

       2. Firearms incident reports per person in Birmingham is low (Ranking around 100). Hence "More firearms incidents per head in Birmingham" claim is False

3. Crimes involving firearms are closely associated with drugs offences - Media

   • Conclusion:
     • We _cannot conclude_ if weapons are related with Drug crimes based on score of 0.5
     • Reason: By looking at the correlation score (0.54) it is correlated but not enough to conclude the statement "Drugs are associated with Weapons" as it did not cross (0.7) correlation threshold
     • There are 5 Areas which have correlation >0.7 indicating areas having weapons are involved in drug related:
       • Areas: Birmingham, East Riding of Yorkshire, Manchester, Salford, Stockport </strong>